Fact checking

ABSTRACT

The present invention relates to a method and system for verification scoring and automated fact checking. More particularly, the present invention relates to a combination of automated and assisted fact checking techniques to provide a verification score. According to a first aspect, there is a method of verifying input data, comprising the steps of: receiving one or more items of input data; determining one or more pieces of information to be verified from the or each item of input data; determining which of the one or more pieces of information are to be verified automatically and which of the one or more pieces of information require manual verification; determining an automated score indicative of the accuracy of the at least one piece of information which is to be verified automatically; and generating a combined verification score which gives a measure of confidence of the accuracy of the information which forms the or each item of input data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityof U.S. application Ser. No. 16/643,567, filed 1 Mar. 2020, which is aU.S. Patent application claiming the benefit PCT InternationalApplication No. PCT/GB2018/052438, filed on 29 Aug. 2018, which claimsthe benefit of U.K. Provisional Application No. 1713817.3, filed on 29Aug. 2017, and U.S. Provisional Application No. 62/551,357, filed on 29Aug. 2017, all of which are incorporated in their entireties by thisreference.

TECHNICAL FIELD

The present invention relates to a method and system for verificationscoring and automated fact checking. More particularly, the presentinvention relates to a combination of automated and assisted factchecking techniques to provide a verification score

BACKGROUND

Owing to the increasing usage of the internet, and the ease ofgenerating content on micro-blogging and social networks like Twitterand Facebook, articles and snippets of text are created on a daily basisat an ever-increasing rate. However, unlike more traditional publishingplatforms like digital newspapers, micro-blogging platforms and otheronline publishing platforms allow a user to publicise their statementswithout a proper editorial or fact-checking process in place.

Writers on these platforms may not have expert knowledge, nor researchthe facts behind what they write, and currently there is no obligationto do so. Content is incentivised by catchiness and that which may earnmost advertising click-throughs (content being optimised in this waysometimes being referred to as “clickbait”), rather than quality andinformativeness.

Therefore, a large amount of content to which internet users are exposedmay be at least partially false or exaggerated, but stillshared/presented as though it were true.

Currently, the only way of verifying articles and statements made onlineis by having experts in the field of the subject matter either approvecontent once it is published or before it is published. This requires asignificant number of reliable expert moderators to be on hand andapproving content continuously, which is not feasible.

Existing methods/systems for automatically verifying content usuallystruggle in complex situations where there are a number of variables tobe considered.

Additionally, existing methods/systems for verifying content which arenot automated are unscalable, costly, and very labour-intensive.

SUMMARY OF THE INVENTION

Aspects and/or embodiments seek to provide a method of verifying andscoring input data by implementing automated and assisted fact checkingtechniques.

According to a first aspect, there is a method of verifying input data,comprising the steps of: receiving one or more items of input data;determining one or more pieces of information to be verified from the oreach item of input data; determining which of the one or more pieces ofinformation are to be verified automatically and which of the one ormore pieces of information require manual verification; determining anautomated score indicative of the accuracy of the at least one piece ofinformation which is to be verified automatically; and generating acombined verification score which gives a measure of confidence of theaccuracy of the information which forms the or each item of input data.

Such a method may verify input data automatically and/orsemi-automatically and generate a verification score, or a “truth”score, that can combine automated content scoring and manualclaim/statement checking.

Optionally, the step of receiving one or more input data comprises atleast one of: automatically identifying input data to be verified;manual submission of information to be verified by a user; and obtainingreference information from one or more information channels.

In some instances, the input data may be received through a mediamonitoring engine. Input data may include media from Twitter, Facebook,blogging websites, and news articles, as well as sentences, articles orparagraphs submitted by a user or users.

Optionally, the reference information is gathered in dependence upon theone or more pieces of information to be verified.

The method may include (the use of) algorithms to automatically obtainreference information that is relevant to the pieces of information tobe verified. The reference information may include facts that can beused to fact check particular claims/statements or articles. As anexample, the reference information can be gathered from open knowledgedatabases of facts or data inputted directly/categorised/verified asfactual information by a user or users.

Optionally, the method further comprises the use of natural languageprocessing techniques and/or other computational methods.

For example, any article, statement or comment can contain a number ofclaims, or statements, which may need to be verified. Quantitativestatements (e.g. the population of London is 12 million people) aregenerally easier to verify compared to qualitative statements (e.g. thepopulation of London is generally less tolerant of delayed publictransport), and techniques such as semantic parsing may be used to breakup the incoming article/statement/comment (input data) and identify thevarious components of the data.

Optionally, the automated score is provided for information comprisingany one of: a sentence, a paragraph, an article and/or a full newsstory.

The method may provide a score for the content in its entirety.

Optionally, the automated score comprises the use of at least oneclassifier modules to identify fake or misleading content. Optionally,the classifier modules comprise any one of: a clickbait detectionmodule; a stance detection module; and content-density module and othermodules as specified below. These classifier modules may be any type ofgeneric supervised or unsupervised machine learning classifiers.

Optionally, the automated score comprises using natural languageprocessing and/or other computational methods to provide a probabilisticscore.

For example, this probabilistic score may be obtained without usingreference information. Rather than verifying a fact against referenceinformation, this method may provide a score using an arbitrary measuresuch as how much of a resemblance to, or appearance of being, clickbaita particular piece of content may be.

Optionally, the automated score is generated in accordance withweightings from the classifier modules.

The weightings assigned to each module may be changed at any point intime.

Weightings for example may be provided for variables in order toautomate a score which is indicative of major factors.

Optionally, further comprising a step of providing a user with afact-checking tool to determine a manual score indicative of theaccuracy of the at least one piece of information requiring manualverification, wherein the manual score is provided by the assistance ofa human fact-checker, wherein the manual score is provided by theassistance of a human fact-checker.

The manual score complements the automated score to provide the overallverification score. The manual score relates to assessing the veracityof individual claims and statements in the input data and is a keycomponent of the verification score.

Optionally, the manual score further comprises detection of one or morestatements from the one or more pieces of information. Optionally, themanual score is provided for information comprising a statement.Optionally, the statement forms part of any one of an online post, aparagraph, an article or a full news story.

Some components of the manual score relate to verifying individualstatements in the body of text. Some of these statements may beautomatically verified and form part of the automated score. In somecases, a claim/statement may be complex and may not be able to beverified automatically by a fact checking method or system. Where theclaims/statements contain a number of variables, and thus complex, itmay need to be verified by a human expert and is deemed to be a manualscore.

Optionally, the manual score further comprises comparing information tobe verified against public databases or reference information.Optionally, the manual score comprises detection of one or morestatements from the one or more pieces of information.

This may provide the expert fact-checkers information to compare aclaim/statement against. Further, the reference information may relateto known factual information for a given topic or subject matter.

Optionally, the step of detecting one or more statements comprisessemantic parsing of the one or more pieces of information.

The semantic parsing of the pieces of information to obtain statementswill be automated, to obtain semantic parses. This may enable acommunity of fact checkers to generate training labels of correctlogical forms for a given semantic parse of a claim/statement.

Optionally, the manual score further comprises at least of one: havingan expert score for each human fact-checker; allocating a claim to themost suitable fact-checker; the use of machine learning to automaticallygather supporting or negating arguments for each claim

from the reference information or a public database; the humanfact-checker providing a counter-hypothesis; the human fact-checkerproviding a counter-argument; the human fact-checker providingstep-by-step reasoning; and providing a reasoned conclusion and/orstatements for the claim being verified.

Along with providing a verification of whether or not a claim/statementis true or false, optionally an explanation as to why theclaim/statement is true or false may also be provided. This can provideclarity of the final verification and gives well-reasoned justificationstogether with each check. This explanation as to why the claim/statementis true or false can form training data for generating an automated factcheck of the statement in the future.

Optionally, the manual score is provided using a fact-checker network orplatform.

The network or platform may be a peer-to-peer network or platform ofexperts.

Optionally, the expert score for each human fact-checker is indicativeof reliability of each human fact-checker. Optionally, the expert scorefor each human fact-checker is determined through an analysis of one ormore of: fact-checker bias; fact-checker credibility; fact-checkerprofile; and/or content generated by the fact-checker.

Optionally, the method comprises providing a manual score orverification indicative of the accuracy of the automated score and/or toadjust the automated score.

In this way, the method may further verify the automated content scoreby have a human fact-checker take a second look.

Optionally, the automated and manual scores are generated upon aweighting analysis performed based on a plurality of factors.

Optionally, the method further comprising the step of storing theverification output on a realtime content quality database.

Optionally, the real-time content database is adapted for a specificuser type.

The truth score may be analysed over time for different writers,authors, domains, people, websites, etc., and be stored and presented asa credibility index.

Optionally, a combination of the automated score and a crowdsourcedscore generates a verification score. In some instances, a scoregenerated from an automated verification and a score generated from acrowdsourced, or semi-automated, verification may be combined to form atruth score for a claim/statement or article.

Optionally, the verification output is present in a form suitable for anapplication program interface, as a graphical representation and/or as acredibility score or trust score.

According to a second aspect, there is provided a method of processingand detecting one or more pieces of information to be verified from oneor more input data, the method comprising; determining which of the oneor more pieces of information are to be verified automatically and whichof the one or more pieces of information require manual verification.

According to a third aspect, there is provided a method of generating averification score, the method comprises; generating a verificationscore which gives a measure of confidence of the accuracy of informationwhich forms one or more input data, the verification score comprising aweighted sum one or more automated scores and one or more manual scores.

According to a fourth aspect, there is provided a method of verifyinginput data, comprising the steps of; receiving one or more items ofinput data; determining one or more pieces of information to be verifiedfrom the or each item of input data; determining which of the one ormore pieces of information are to be verified automatically and which ofthe one or more pieces of information require manual verification;determining an automated score indicative of the accuracy of the atleast one piece of information which is to be verified automatically;providing a user with a fact checking tool to determine; a first manualscore indicative of the accuracy of the at least one piece ofinformation requiring manual verification; and a second manual scoreindicative of the accuracy of a combination of the or each automatedscore; and generating a verification score which gives a measure ofconfidence of the accuracy of the information which forms the or eachitem of input data, wherein the verification score comprises; acombination of the or each automated score and the or each first manualscore; or a combination of the or each automated score and the or eachsecond manual score.

According to a fifth aspect, there is provided an apparatus operable toperform the method of any preceding feature.

According to a sixth aspect, there is provided a system operable toperform the method of any preceding feature.

According to a seventh aspect, there is provided a computer programoperable to perform the method and/or apparatus and/or system of anypreceding feature.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described, by way of example only and withreference to the accompanying drawings having like-reference numerals,in which:

FIG. 1 illustrates a fact checking system;

FIG. 1 a is an expanded section of FIG. 1 , more specifically detailingvarious claim channels and the automated misleading content detectionalgorithm;

FIG. 1 b is an expanded section of FIG. 1 , more specifically detailingthe different claim groups within the system;

FIG. 1 c is an expanded section of FIG. 1 , more specifically detailingoutputs of the fact checking network and platform and various productlines;

FIG. 2 illustrates an automated content scoring module of a factchecking system;

FIG. 3 illustrates a flowchart of truth score generation including bothmanual and automated scoring;

FIG. 4 illustrates the manual scoring module of a fact checking systeminvolving human fact-checkers; and

FIG. 5 shows an example of a claim tagging user interface for annotatingclaims.

SPECIFIC DESCRIPTION

Embodiments will now be described with the assistance of FIGS. 1 to 5 .

It is proving increasingly difficult for users to detect bias and judgefor provenance and quality in content that users are exposed to. Therise of user-generated content has resulted in a considerable amount ofcontent online being produced without fact checking standards oreditorial policy, so judging conformity to such a policy is needed toempower any reader of content to judge the truth of such content. Thus,there is an unprecedented need for a truth “layer” or similar overlay oncontent that can identify such content as being fake, misleading orfalse, and then verify the claims and information made in the contentitself.

Illustrating an example embodiment of a fact checking system overview isFIG. 1 . FIG. 1 shows a flowchart of the fact checking system startingfrom a media monitoring engine 101. Media content/online informationfrom claim channels 102 such as UGC, reputable sources,

rumour aggregators, official sources and market participants arecollected and input into an automated misleading content detectionalgorithm 103. Claim channels 102 are not limited to the specificexamples aforementioned or examples detailed in FIG. 1 a and someembodiments may include content provided by other sources.

As depicted in FIG. 1 , the automated misleading contentalgorithm/detector 103 forms the first major part of the system. Thismodule takes in content from several media monitoring systems 101 andanalyses it based on various natural language processing techniques foridentifiers that it could be fake or misleading, as well as generallyscoring the content for its quality. In short, these may include:

-   -   1. The original domain and I P address of the news article and        whether it may be produced and distributed by a bot network        based on pattern analysis, or whether it is a clear copy of a        real and trusted domain with a modification.    -   2. Weighted classification to suspect an article, including        missing citations, author names, ‘about us’ section, spelling        errors, out of context quotes, one-sidedness, outrageousness.    -   3. Crowdsourced data on article trustworthiness and/or any other        characteristic in relation to the content.    -   4. A comparison of the headline to the article body for        “clickbait” detection.    -   5. Identification of how “clickbait” the headline is.    -   6. The stance of reputed news agencies to the article: support,        agree, disagree, discuss, unrelated.    -   7. And other methods.

As illustrated in FIG. 1 a , the automated misleading content detectionalgorithm 103 consists of various analysation techniques. Namely,analysing historical credibility, consistency/stance detection,references within claims, language analysis e.g. linguistic and semanticanalysis, metadata, bias, clickbait detection and content density.

A claim detection system may be present in a fact checking systemwhereby it deploys annotated claims fromexperts/journalists/fact-checkers. An example claim annotation system byFull Fact is shown in FIG. 5 . In this example embodiment, Briefr isused. You can see a user generated “citation needed” tag which leads tothe claim label as shown as 501.

The next phase is to develop a specific workflow for a citation neededtag. In order to suggest the type of claim it would require an actionsuch as a click, a comment(s) may be inputted, and an evidence link/URLmay be provided via the browser extension. In this way, the workflowprocess retrieves various data such as:

-   -   What is the claim;    -   Explanation/fact check/comment for that claim;    -   Evidence which would be submitted as a link for that claim; and    -   Counter-claim (the sentence that contradicts the claim or gives        a different view of the facts), the machine may or may not        extract automatically from the evidence URL, or the user may        input the evidence into a text box.

In an example embodiment, a claim detector 104 may be present to detect,parse and cluster claims. A claim filter 105 may also be present whichgroups claims into separate categories as shown in FIG. 1 b. Accordingto at least one embodiment, in the case of a complex claim (one that thesystem cannot automatically verify), the assistance of a human factchecker is needed. This is referred to as humans-in-the-loop. Forexample, claim groups may include:

-   -   1) Instant and binary: Automatically verifiable against public        databases and will result in a very high confidence true/false        outcome.    -   2) Instant and probabilistic: Assessable using NLP and other        computational methods but no hard facts to verify against. This        may result in a multi-dimensional continuum of likelihoods        between true and false.    -   3) Human-in-the-loop and binary: Verifiable against public        databases but needs research/check/input by an expert analyst.        The confidence outcome may be similar to that of an instant and        binary claim group.    -   4) Human-in-the-loop and probabilistic: Truth locked on private        database/inaccessible due to legal/other constraints, or no real        facts to verify against rumours/event.

Content clustered into one or more human-in-the-loop claim groups can beinput into a fact checking network and platform 109 where experts invarious domains provide machine readable arguments in order to debunkclaims. In this way, the community is self-moderating in order to ensurethe best fact-check receives the highest reward. The fact checkingnetwork and platform 109 may be used for annotation purposes and mayalso be combined with other tools/platforms, for example a bias scoringplatform.

FIG. 1 also illustrates example outcomes from inputting content fromvarious claim channels 102 to an automated misleading content detectionalgorithm 103, a claim detector 104 and a claim filter 105. The outcomesare more specifically described in FIG. 1 c, and these include, but arenot limited to, as shown as 110, the following:

-   -   1) Content moderation on demand: moderation of any media stream        for fake news.    -   2) Tracking abusive users: The ability to blacklist more and        more bad actors.    -   3) Determining a probability truth score: Assigning a score to a        claim which may be added to a source track record.    -   4) Determining source credibility: Updating track records of        sources of claims.    -   5) Annotations by real-time expert analysts: Rating provided by        experts in various domains in order to debunk content.    -   6) Providing alternative viewpoints: No claim or rumour is taken        for granted and has additional viewpoints.

FIG. 2 depicts an “Automated Content Scoring” module 206 which producesa filtered and scored input for a network of fact checkers. Input intothe automated content scoring module 206 may include customer contentsubmissions 201 from traders, journalists, brands, ad networks useretc., user content submissions 202 from auto-reference andclaim-submitter plugins 216 and content identified by the mediamonitoring engine 101. The content moderation network of fact checkers207 including fact checkers, journalists, verification experts, groupedas micro taskers and domain experts, then proceeds by verifying thecontent as being misleading and fake through an Al-assisted workbench208 for verification and fact-checking. The other benefit of such asystem is that it provides users with an open, agreeable quality scorefor content. For example, it can be particularly useful for newsaggregators who want to ensure they are only showing quality content buttogether with an explanation. Such a system may be combined with orimplemented in conjunction with a quality score module or system.

This part of the system may be an integrated development environment orbrowser extension for human expert fact checkers to verify potentiallymisleading content. This part of the system is particularly useful forclaims/statements that are not instantly verifiable, for example ifthere are no public databases to check against or the answer is toonuanced to be provided by a machine. These fact checkers, as experts invarious domains, have to carry out a rigorous onboarding process, anddevelop reputation points for effectively moderating content andproviding well thought out fact checks. The onboarding process mayinvolve, for example, a standard questionnaire and/or based on profileassessment and/or previous manual fact checks made by the profile.

Through the Al-assisted workbench for verification and fact-checking208, a per-content credibility score 209, contextual facts 210 andsource credibility update 211 may be provided. The source credibilityupdate may update the database 212 which generates an updatedcredibility score 213 and thus providing a credibility index as shown as214 in FIG. 2 . Contextual facts provided by the Al-assisted userworkbench 208 and credibility scores 213 may be further provided as acontextual browser overlay for facts and research 215.

Real-time content quality and fact check databases 108 and 111 are usedto store data for training algorithms as well as to determine a qualityfact check and are used to enhance the system's automated fact checkingcapabilities. The data within the real-time content quality database maybe delivered to users e.g. clients 114. On the other hand the real-timefact check database in provided to product lines 113, for example APIaccess, human-facing dashboard and content trust seal.

In embodiments algorithms are substantially domain-adaptable, given thatusers may provide data from a variety of sites (social sites, news,political blog posts, lifestyle blogs, etc). For that, data isaggregated from the various sources and stratified sampling may beimplemented to build the training and the test datasets. The finalperformance metric may be based on a test dataset that encompasses avariety of sources. In terms of process, datasets are gathered from opensources or from research papers. After carrying out error analysis, oneor multiple annotation exercises are run on a sample of customer data,and which is used to re-train the model.

In terms of the data, various annotation exercises can be implemented onboth crowdsourcing platforms (Crowdflower, Mechanical Turk) and otherexpert annotation platforms such as BriefR.

Two of the main challenges of the ML models are making sure the modelsare fair and up-to-date. News stories and threats keep changing everyday, and it is necessary to be able to detect new content. For example,for models to be “fair” and not too biased (i.e. only detectingright-wing stories as hyperpartisan content), it is required to makesure that the training data has been collected from a balanced set ofannotators which is representative from the set of views we would liketo incorporate into our models. In order to achieve these two goals, aunique set of communities of experts/users will provide thehuman-in-the-loop in order to annotate new trending stories etc. Modelsmay be used to identify the top toxic trending stories, which will thenbe given to annotators to remove false positives. Then, in order toincrease the recall of classifiers within the system, data which isdirectly reported/flagged by our communities of experts, as well as takean unsupervised approach to find the top trending themes will be used.In some embodiments, stories/content coming from both the supervisedapproach and unsupervised, will be then fed to experts/annotators as afinal check to get a

labelled set of toxic stories. These articles/pieces of content can thenbe fed back into the machine learning (ML) models to re-train them aswell as updating fact checks for content.

The assisted fact checking tools have key components that effectivelymake it a code editor for fact checking, as well as a system to build adataset of machine readable fact checks, in a very structured fashion.This dataset will allow a machine to fact check content automatically invarious domains by learning how a human being constructs a fact check,starting from a counter-hypothesis and counter-argument, an intermediatedecision, a step by step reasoning, and a conclusion. Because the systemcan also cluster claims with different phrasings or terminology, itallows for scalability of the system as the claims are based online(global) and not based on what website the user is on, or which websitethe input data/claim is from. This means that across the internet, ifone claim is debunked it does not have to be debunked again if it isfound on another website.

In an embodiment, a user interface may be present wherein enablingvisibility of labels and/or tags, which may be determined automaticallyor by means of manual input, to a user or a plurality of users/expertanalysts. The user interface may form part of a web platform and/or abrowser extension which provides users with the ability to manuallylabel, tag and/or add description to content such as individualstatements of an article and full articles.

FIG. 3 illustrates a flowchart of truth score generation 301 includingboth manual and automated scoring. A combination of an automated contentscore 302 and a crowdsourced score 303 i.e. content scores determined byusers such as expert annotators, may include a clickbait score module,an automated fact checking scoring module, other automated modules, userrating annotations, user fact checking annotations and other userannotations. In an example embodiment, the automated fact checkingscoring module comprises an automatic fact checking algorithm 304provided against reference facts. Also, users may be provided with anassisted fact checking tool/platform 305. Such tool/platform may assista user(s) in automatically finding correct evidence, a task list,techniques to help semantically parse claims into logical forms bygetting user annotations of charts for example as well as otherextensive features.

According to at least one embodiment, in the case of a complex claim(one that the system cannot automatically verify), the assistance of ahuman fact checker is needed. This is referred to as humans-in-the-loop.This embodiment works in the following manner and is depicted in FIG. 4:

-   -   1. Understands and documents the expertise of various fact        checkers on the networks e.g. who is better than others at        checking claims about economic statistics. There may be provided        an expert score 401 to identify such experts.    -   2. Allocates the correct claim or news content to the right fact        checker 402.    -   3. Clusters similar news pieces, rumours or claims together, as        shown as 403, so there is no repetition of fact checking.    -   4. Generates probabilistic content scoring of claims or articles        using an automated algorithm.    -   5. Asks the fact checker to record in a very specific way their        methodology to fact check. By way of an example, this may        include information such as: what were the supporting or        negating sources, what is their counter-claim or hypothesis,        what is the argument they put forth, what is the intermediate        logic of their fact-check, and what is the conclusion or score        they assign to a piece of content. The system will also        incorporate machine learning to assist the fact checkers. For        example, auto-correct certain claims made in the text, surface        the right sources to check particular claims, automatically        gather supporting or negating arguments from the web via stance        detection. There may be provided upvote/downvote data based on        argument quality as shown as 404.    -   6. Has a decision support mechanism, potentially via the        blockchain, to have multiple fact-checkers confirm or support a        fact-check, with different weightings assigned to each        fact-checker for their expertise.    -   7. Produce a stamp or certificate for each piece of content.    -   8. As the system processes more information, the system can        recognise patterns to automatically fact check particular pieces        of content or produce conclusions automatically with a        probabilistic degree of likelihood and produce a truth score        405.

This embodiment provides reasoning as to why the content is misleadingand provides some information to the fact checker network to indicatewhy this content has been flagged, as opposed to providing theinformation with no context as to what is to be checked. In this way,the system may provide an explainable aspect to the assessed orfact-checked information. In FIG. 1 , there is illustrated anexplainable content quality score 107.

Importantly, the weights of the automated content scores may beadjusted. For example, “clickbait” refers to a method of obtaining aninterest from a user in an article, generally using a sensationalist orhighly exaggerated headline. A user clicks on what appears to be a veryinteresting or informative article, which, usually, does not live up toexpectations. The terms “clickbaitedness” or “how clickbait (is this)”may refer to a level/quality of or resemblance to “clickbait” detectedas part/in of an article. A semi-automated or ‘assisted’ fact-checkingarrangement involving human checking methods to assign the same score tocontent, may be provided. Such a semi-automated arrangement may includeany of the preceding algorithmic methods, and/or one or more of:

-   -   1. How many claims are referenced in the article with a link;    -   2. How many claims are fact-checked as true vs. unverified        claims; and/or    -   3. How many claims are from first hand, secondary or tertiary        sources and how many claims are sourced from somewhere else.

Thus, if the expert fact checker believes that “clickbaitedness” is amore accurate indicator of what is to be flagged to their contentmoderation network to enhance recall of misleading content, thisweighting factor of the automated content score may be increasedaccordingly. This embodiment includes some elements of explainablemachine learning where it can explain and account for its reasoning offact checking.

This system and method is significantly cheaper and faster than havingindividual teams of fact-checkers, because it harnesses knowledge andwisdom of the crowd to self-correct and manage. The system is configuredto allocate work segments efficiently amongst a network as opposed tobeing unallocated. The system may also provide different weightings todifferent people implicitly in order to fact check different contentbased on their expertise, to make conflict resolution faster thanWikipedia (for example) and in order to reach conclusions more quickly.Also, the platform may take in any form of content, even videos, and isnot limited to one network (e.g. Wikipedia articles).

The fact-checking expert network may explicitly encourage experts indifferent fields to correct the claims and databases. The best factchecks on the platform get upvoted by other experts and/or the publicand thus people are encouraged to write better quality arguments. Thisenables a real-time, on demand network of the top minds or experts incertain fields to be built, in order to dispel rumours and mistruthsabout things.

Another potential benefit of some embodiments providing such a platformcan be that users have the ability to see opinions on things that arefar beyond their ‘filter bubbles’ (that is to say what they may notusually see) on social media platforms. People associated with bothsides of a story may have the ability to fact check a rumour or claim,and only the rumour with the best argument ‘wins’ in terms of being ableto fact-check a statement with evidence. In this way, producers of thecontent in question are motivated and encouraged to create higherquality content.

The system may also pick up potential fake news, rumours and hoaxesfaster than other systems, to achieve close to a real-time solution.This is because content is fed into the network in real-time rather thanwhen the rumour or news has picked up attention or popularity. Thesystem may be combined or implemented in conjunction with other systemsor methods for determining, for example, bias such ashyper-partisanship, content and/or author credibility and/orcontentiousness within content. The system may further be integrated toprovide an explanation and be provided as part of an annotation platformwhich may be taken into account within annotator profiles. In an exampleembodiment, there may be provided a web based method, systems andalgorithms which determine automatically and/or semi-automatically thelevel of bias/hyper-partisanship and/or scores in relation tocredibility/truthfulness.

Due to the ability to perform claim clustering and rumour similaritytechniques, if a web rumour is available on a website and is the same asthe check that has been done on a different website, the system canblanket the web with the fact checks for a particular rumour and combinethem, rather than having an expert network start from scratch in theirchecks.

By using machine learning techniques, the system will slowly start toassist and aid human fact-checker networks to check things faster andmore efficiently across the web, because it will start to record howhumans do the same task so that a machine can start to help in this verytime consuming and complex process.

This semi-automated system for fact-checking and content scoring can beavailable for use for multiple industries and use cases, including:

-   -   1. Hedge funds who want to verify signals they see in news as        being real vs. fake, and have a reliability score to them.    -   2. Platforms such as Facebook or Google that need to clean up        their networks of fake news.    -   3. News aggregators and any platforms that host links that need        to provide their users with accurate news feeds.    -   4. Advertising network inventory owners who own link inventories        for advertisers to advertise on and want to ensure all content        is clean and can reflect brand image.

Another set of algorithms and/or similar technology may be used toautomatically assist a fact-checking network with their fact checkingprocess of verifying individual claims/statements, which may lead to acrowdsourced score. Such assistance may include one or more of:

-   -   detecting assertions, rumours and/or claims in bodies of text        using machine learning methods (which may include neural        networks), the said assertions, rumours, and/or claims requiring        fact-checking;    -   helping a user find reference sources against which to        fact-check (which, for example, may include a claim about        economic growth presented alongside a link to a World Bank data        source, against which to check, with the correct country and        date filled in); splitting media data into clusters of        viewpoints, and for the same story, stories that are for or        against a target individual or claim in nature;    -   assessing the provenance of the headline, including who was the        original reporter of the story, for example a Tweeter or the        Associated Press;    -   starting to provide, automatically, a task list for a fact        checker for any given claim or rumour, in terms of the steps to        take to check it which may be different based upon the content        of the claim;    -   providing alternative sources for each topic in the body of an        article, and additional context including graphics, further        reading and so on;    -   assessing how much text has been copied from another article        that is already known about;    -   assessing information about the author and/or persons identified        in the story;    -   identifying quotations which have been misquoted from their        original quotes in source material;    -   providing a button for a fact-checker to open, automatically, a        set of tabs on their browser pre-searched with the key terms;    -   providing a fact checker with a chart or table from a factual        source appropriate to the fact they should be checking;    -   Providing a fact checker with the correct link to visit to fact        check content; and/or allowing a fact checker to score the        content from 1-10.    -   Providing an indication or score based on other systems such as        bias, hyper-partisan and contentiousness etc.

In these ways, a score may effectively be crowdsourced.

Machine learning is the field of study where a computer or computerslearn to perform classes of tasks using the feedback generated from theexperience or data gathered that the machine learning process acquiresduring computer performance of those tasks.

Typically, machine learning can be broadly classed as supervised andunsupervised approaches, although there are particular approaches suchas reinforcement learning and semi-supervised learning which havespecial rules, techniques and/or approaches.

Supervised machine learning is concerned with a computer learning one ormore rules or functions to map between example inputs and desiredoutputs as predetermined by an operator or programmer, usually where adata set containing the inputs is labelled.

Unsupervised learning is concerned with determining a structure forinput data, for example when performing pattern recognition, andtypically uses unlabelled data sets. Reinforcement learning is concernedwith enabling a computer or computers to interact with a dynamicenvironment, for example when playing a game or driving a vehicle.

Various hybrids of these categories are possible, such as“semi-supervised” machine learning where a training data set has onlybeen partially labelled. For unsupervised machine learning, there is arange of possible applications such as, for example, the application ofcomputer vision techniques to image processing or video enhancement.Unsupervised machine learning is typically applied to solve problemswhere an unknown data structure might be present in the data. As thedata is unlabelled, the machine learning process is required to operateto identify implicit relationships between the data for example byderiving a clustering metric based on internally derived information.For example, an unsupervised learning technique can be used to reducethe dimensionality of a data set and attempt to identify and modelrelationships between clusters in the data set, and can for examplegenerate measures of cluster membership or identify hubs or nodes in orbetween clusters (for example using a technique referred to as weightedcorrelation network analysis, which can be applied to high-dimensionaldata sets, or using k-means clustering to cluster data by a measure ofthe Euclidean distance between each datum).

Semi-supervised learning is typically applied to solve problems wherethere is a partially labelled data set, for example where only a subsetof the data is labelled. Semi-supervised machine learning makes use ofexternally provided labels and objective functions as well as anyimplicit data relationships. When initially configuring a machinelearning system, particularly when using a supervised machine learningapproach, the machine learning algorithm can be provided with sometraining data or a set of training examples, in which each example istypically a pair of an input signal/vector and a desired output value,label (or classification) or signal. The machine learning algorithmanalyses the training data and produces a generalised function that canbe used with unseen data sets to produce desired output values orsignals for the unseen input vectors/signals. The user needs to decidewhat type of data is to be used as the training data, and to prepare arepresentative real-world set of data. The user must however take careto ensure that the training data contains enough information toaccurately predict desired output values without providing too manyfeatures (which can result in too many dimensions being considered bythe machine learning process during training, and could also mean thatthe machine learning process does not converge to good solutions for allor specific examples). The user must also determine the desiredstructure of the learned or generalised function, for example whether touse support vector machines or decision trees.

The use of unsupervised or semi-supervised machine learning approachesare sometimes used when labelled data is not readily available, or wherethe system generates new labelled data from unknown data given someinitial seed labels.

Machine learning may be performed through the use of one or more of: anon-linear hierarchical algorithm; neural network; convolutional neuralnetwork; recurrent neural network; long short-term memory network;multi-dimensional convolutional network; a memory network; or a gatedrecurrent network allows a flexible approach when generating thepredicted block of visual data. The use of an algorithm with a memoryunit such as a long short-term memory network (LSTM), a memory networkor a gated recurrent network can keep the state of the predicted blocksfrom motion compensation processes performed on the same original inputframe. The use of these networks can improve computational efficiencyand also improve temporal consistency in the motion compensation processacross a number of frames, as the algorithm maintains some sort of stateor memory of the changes in motion. This can additionally result in areduction of error rates.

Any system feature as described herein may also be provided as a methodfeature, and vice versa. As used herein, means plus function featuresmay be expressed alternatively in terms of their correspondingstructure.

Any feature in one aspect may be applied to other aspects, in anyappropriate combination. In particular, method aspects may be applied tosystem aspects, and vice versa. Furthermore, any, some and/or allfeatures in one aspect can be applied to any, some and/or all featuresin any other aspect, in any appropriate combination.

It should also be appreciated that particular combinations of thevarious features described and defined in any aspects can be implementedand/or supplied and/or used independently.

1. (canceled)
 2. A method comprising: determining an automated scoreindicative of an accuracy of a first set of pieces of information, ofone or more items, to be verified by automatic verification; providing auser with a fact-checking tool to determine a manual score indicative ofthe accuracy of a second set of pieces of information, of the one ormore items, to be verified by manual verification, the fact-checkingtool being configured to find correct evidence for the second set ofpieces of information and to present a task list for a humanfact-checker to check the second set of pieces of information, themanual score being provided by assistance of the human fact-checker; andcombining the automated score and the manual score to generate acombined verification score, the combined verification score giving ameasure of confidence of the accuracy of the information which forms theone or more items.
 3. The method of claim 2, wherein the first set ofpieces of information comprising at least one of a sentence, aparagraph, an article, or a full news story.
 4. The method of claim 2,wherein the determining of the automated score comprises using at leastone classifier module to identify fake content or misleading content. 5.The method of claim 4, wherein the at least one classifier modulecomprises at least one of: a clickbait detection module; a stancedetection module; or a content-density module.
 6. The method of claim 2,wherein the determining of the automated score comprises using at leastone of natural language processing or another computational method togenerate a probabilistic score.
 7. The method of claim 2, wherein theautomated score is determined in accordance with one or more weightingsfrom multiple classifier modules.
 8. The method of claim 2, wherein thesecond set of pieces of information comprising a statement, and whereinthe statement forms part of any one of an online post, a paragraph, anarticle, or a full news story.
 9. The method of claim 2, wherein thedetermining of the manual score comprises comparing a piece ofinformation, from the second set of pieces of information, against oneor more public databases or against reference information.
 10. Themethod of claim 2, wherein the determining of the manual score comprisesdetecting one or more statements from one or more pieces of the secondset of pieces of information.
 11. The method of claim 10, wherein thedetecting of the one or more statements comprises semantic parsing ofthe one or more pieces.
 12. The method of claim 2, wherein thedetermining of the manual score comprises: accessing an expert score foreach human fact-checker; allocating a claim to a most suitablefact-checker based on an accessed expert score of the most suitablefact-checker; and using machine learning to automatically gather, forthe most suitable fact-checker, one or more supporting or negatingarguments for the claim from at least one of reference information or apublic database.
 13. The method of claim 12, wherein the determining ofthe manual score comprises: receiving from the most suitablefact-checker at least one of a counter-hypothesis for the claim, acounter-argument for the claim, a fact-checker step-by-step reasoningfor the claim; and providing a reasoned conclusion or statements for theclaim based on the at least one of the counter-hypothesis for the claim,the counter-argument for the claim, or the fact-checker step-by-stepreasoning for the claim from the most suitable fact-checker.
 14. Themethod of claim 12, wherein the expert score for each human fact-checkeris indicative of reliability of each human fact-checker.
 15. The methodof claim 12, wherein the expert score for each human fact-checker isdetermined through an analysis of at least one of fact-checker bias,fact-checker credibility, fact-checker profile, or content generated bythe human fact-checker.
 16. The method of claim 2, comprising storingthe combined verification score on a real-time content quality database.17. The method of claim 16, wherein the real-time content qualitydatabase is adapted for a specific user type.
 18. A computer programoperable to perform operations comprising: determining an automatedscore indicative of an accuracy of a first set of pieces of information,of one or more items, to be verified by automatic verification;providing a user with a fact-checking tool to determine a manual scoreindicative of the accuracy of a second set of pieces of information, ofthe one or more items, to be verified by manual verification, thefact-checking tool being configured to find correct evidence for thesecond set of pieces of information and to present a task list for ahuman fact-checker to check the second set of pieces of information, themanual score being provided by assistance of the human fact-checker; andcombining the automated score and the manual score to generate acombined verification score, the combined verification score giving ameasure of confidence of the accuracy of the information which forms theone or more items.
 19. The computer program of claim 18, wherein thedetermining of the manual score comprises: accessing an expert score foreach human fact-checker; allocating a claim to a most suitablefact-checker based on an accessed expert score of the most suitablefact-checker; and using machine learning to automatically gather, forthe most suitable fact-checker, one or more supporting or negatingarguments for the claim from at least one of reference information or apublic database.
 20. The computer program of claim 19, wherein thedetermining of the manual score comprises: receiving from the mostsuitable fact-checker at least one of a counter-hypothesis for theclaim, a counter-argument for the claim, a fact-checker step-by-stepreasoning for the claim; and providing a reasoned conclusion orstatements for the claim based on the at least one of thecounter-hypothesis for the claim, the counter-argument for the claim, orthe fact-checker step-by-step reasoning for the claim from the mostsuitable fact-checker.
 21. A system comprising: a media monitoringengine configured to receive one or more items of input data; anautomated misleading content detection algorithm configure to determineone or more pieces of information to be verified from the input data; aclaim detector configured to determine a first set of pieces ofinformation to be verified automatically and a second set of pieces ofinformation to be verified manually; a set of classifier modulesconfigured to determine an automated score indicative of accuracy of atleast one piece of information in the first set; a fact-checking toolconfigured to provide a user with a fact-checking tool to determine amanual score indicative of the accuracy of at least one piece ofinformation in the second set of pieces of information, thefact-checking tool being configured to find correct evidence for thesecond set of pieces of information and to present a task list for ahuman fact-checker to check the second set of pieces of information, themanual score being provided by assistance of the human fact-checker; anda truth score generator configured to combine the automated score andmanual score to generate a combined verification score, the combinedverification score giving a measure of confidence in the accuracy of theinformation in the input data.